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ABSTRACT 



A 5 -month project was conducted by the Greater Pittsburgh 
Literacy Council (GPLC) to develop a better posttest to evaluate student 
goals and their tutoring experience after they had worked with a volunteer 
tutor for approximately 50 hours. During the project, the existing evaluation 
form was reviewed, and a new one-page posttest evaluation sheet was created. 
It was hoped that the student answers collected from the new form would be 
longer and more in-depth and that they would reveal more information about 
students' achievements and completed goals, their future goals, and their 
problems with their tutors or any other obstacles preventing them from 
completing the literacy program. Four area coordinators used the new form for 
posttesting during the project. Results of the project showed that the new 
form elicited a greater quantity and higher quality of answers when the 
students were asked about their accomplishments and whether they now had any 
new goals. The form was less successful in uncovering problems in the 
tutoring sessions or in the students' lives, in part because the questions 
were not open-ended enough. The project concluded that the form is an 
improvement but still needs further refinement with more input from other 
staff members. (KC) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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Pennsylvania Action Research Monograph 



Note: Action Research is a process of systematic inquiry credited 
to Kurt Lewin who popularized it in the U.S. in the 1940"s. Today 
it is considered a system of qualitative research. Typical of 
action research, none of the individual projects in this monograph 
series claims to have generalizable application beyond the 
specific project described. However, each monograph report can 
serve to be illustrative, instructive and provides the potential 
for replication in other locations. For a level of 
generalizability, it is recommended that the reader seek common 
patterns in the monograph reports in this series, and the wider 
literature, or contact the Action Research Network for assistance 
in this. 




I. ABSTRACT 



The Greater Pittsburgh Literacy Council is an adult basic education program which teaches 
ABE, GED and English as a second language to adults 18 and over. As each student enters our 
program they are tested individually and their goals are discussed at length. The students are then 
matched with a volunteer tutor to work on their educational goals. All students are re-tested after 30 
to 50 hours of tutoring. At the post-test they are asked a series of questions regarding their goals 
and their tutoring experience. 

This project was to change the questions on the Post-test Evaluation Form and to use it for 
a period of five months. The answers on the new form were compared to the answers given on the 
old form during the same period last year. Then the quality and quantity of the answers were 
compared. 

I was attempting to determine the following: 

Will changing the questions on the post-test evaluation sheet, over a four month period, improve 
the quality and quantity of the student responses regarding completed goals and potential concerns. 

My results revealed a greater quality and quantity of answers when the students were asked 
about their accomplishments and if they now had any new goals. My two other goals, to uncover 
problems in the tutoring sessions and the students' lives, to prevent student drop out, were not as 
successful. The questions were not open-ended enough and the students did not reveal tutoring or 
personal problems to the interviewer. 

I believe that while some of the questions on the form are good and can remain, the form 
will need to be revised a second and possibly a third time with more input from other staff 
members. 



II. PROBLEM 

The Greater Pittsburgh Literacy Council is a non-profit agency in the field of adult literacy 
education. This includes, adult basic education (ABE), General Educational Development (GED), 
English as a second language (ESL), family literacy, and workplace literacy. GPLC has a paid 
professional staff of approximately 30 people. GPLC services all of Allegheny County, with one 
main office in East Liberty and 8 area offices throughout Allegheny County. The area offices are 
staffed by one person (Area Coordinator) and serve a specific territory which is broken down by 
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zip codes. I serve the territory of North-West Pittsburgh which covers the broad area of the North 
Side, downtown Pittsburgh, the airport area and the west side of Pittsburgh. 

With the exception of the workplace and family literacy programs, 90% of the adult 
students who enter GPLC's program are tutored by adult volunteers. The remaining 10% are 
tutored in small classes or by paid staff. Prior to tutoring, each volunteer must attend a 12 hour 
tutor training and each student must attend a 1 1/2 hour orientation and an individual evaluation. 

As an area coordinator it is my role to deal with all the volunteers and students who live or 
want to tutor in the territories I cover. I pre- and post-test the students, interview tutors, and make 
tutor I student matches. I also handle all the paper work and problems associated with this, while 
providing ongoing support the tutors and students. By working in the community I serve I can 
meet with tutors and students and be easily accessible to them. 

All students entering the GPLC program are evaluated individually with a mix of formal 
and informal testing. Students are then reevaluated after 30 to 50 hours of tutoring. At each post- 
test students are asked a short series of questions about changes in their reading and it's effect on 
their home and work lives, their tutoring sessions and their goals. The problem is in the questions 
themselves. GPLC is in need of better information on the impact of our program on students lives 
and if the students are accomplishing their goals. We also want to uncover any problems in the 
tutoring sessions themselves and improve future sessions. The evaluation form as it is currently 
written is not eliciting the needed responses from the students. 

As an area coordinator it is my job to complete state forms on all students. On those forms 
we want to be able to report goals the students have and are achieving. Student retention is also an 
ongoing problem. We want to retain students until they student accomplish their stated goals. My 
goal with this project was to improve GPLC's program and my practice by discovering what 
students have accomplished and what goals they have competed. We would hopefully impact 
retention by finding out if there are problems in the tutoring sessions which we could address and 
correct. I also hoped to improve student retention by identifying a personal problem before it 
causes the student to drop out of the program. We then hoped to be able to help the student find the 
appropriate solution. Lastly I hoped in the long run to improve the state forms by reporting more 
completed student goals. 
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III. PLANNING 

My intervention was to create a new, one-page, post-test evaluation sheet. I then asked 
three other Area Coordinators to use this form in place of their old post-test evaluation form during 
every post-test they did in a five month period. I too used this form in my own area. The project 
ran from November 1, 1998 to March 31, 1999. 

The only materials required were the new form and an evaluation of the form at the end of 
the five months. The data was collected by having each coordinator give me copies of all completed 
forms during the five months. The completed forms were then compared to the post-test evaluation 
forms which had been completed during the same five months last year. In other words, I 
compared the answers on the new post-test evaluation form, from November 1, 1998 to March 31, 
1999 to the answers on the old post-test evaluation form from November 1, 1997 to March 31, 
1998. 

Approval was needed from the agency Program Manager who is my direct supervisor and 
it was received. The only constraints to the project was a concern that not enough post-tests were 
completed in this five month period to truly evaluate the form, but twenty-seven forms were 
completed during this time which gave a solid basis for analysis. 

The project was to be considered a success if the answers received from the students on the 
evaluation sheet are longer, more in depth and reveal more information about their achievements 
and completed goals, their future goals and their problems with their tutors and any obstacles 
preventing them from completing our program. 

THE PROBLEM STATEMENT IS: 

Will changing the questions on the post-test evaluation sheet over a four month period, improve the 
quality and quantity of student responses regarding competed goals and potential concerns. 

IV. ACTION 

To complete this project I first consulted with a number of other area coordinators to obtain 
their input on what types of questions they felt the form should have. I then took those suggestions 
and created a new Post-Test Evaluation Form (Appendix II) to replace the old Post-test Evaluation 
Form (Appendix I). 
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I then approached four area coordinators with large areas and asked them to use the form 
beginning on November 1, 1998 and ending on March 31, 1999. The form was to be used during 
every post-test given during this time. 

Out of the ten area coordinators in the program I choose three and myself to implement the 
project. I picked these areas for several reasons. First, they were some of the largest territories and 
consequently were doing a number of post-tests. Secondly, the coordinators themselves, rather 
than volunteers were giving the majority of the post-tests for their area, which I felt would maintain 
consistency in the project. And lastly, these coordinators tended to be thorough in their approach to 
testing and could be relied upon to complete the form each time. 

When the project began I was working in an office in the West End and handling only that 
territory. I had asked the areas of McKeesport/Mon Valley, East Liberty and North Side to work 
with this form. In December of 1998, the North Side Coordinator was promoted and GPLC 
combined the West End and the North Side into one office. That meant that although there would 
be no fewer post-tests given there would be one less objective person to evaluate the form at the 
end of the project. Consequently, when the project ended in March I had only two people to 
evaluate the form. It was suggested by the PAARN facilitator that the other people who had seen 
this form would be GPLC's Reading Specialist and Education Specialist. I then asked them both to 
review the form based on the testing materials they had looked at and their familiarity with the old 
post-test form. This provided two more opinions on the form and another perspective on it. 

V. RESULTS 

In reviewing the success of the new Post-Test Evaluation form I looked at four different 
areas. In two of these areas I was able to directly compare the old Post-Test Evaluation data to the 
new form and obtain hard data and actual percentages. Of the other two areas reviewed, the 
comparison of one area was fairly objective and the other area had no direct comparison on the old 
form. 

The first area reviewed was student goal achievement. I was hoping the new form would 
elicit more student accomplishments. The old form addressed this area with three different 
questions. They were, "In what ways do you think your reading has changed since you started 
with your tutor?", "Have you reached any of your own personal reading goals? Please explain and 
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be specific as you can." and "Are your lessons with your tutor benefiting you an a day to day basis 
either at home or on your job? Please explain." 

I felt the biggest problems with these questions began with the word "reading". Although 
GPLC is a literacy program, when students are pre-tested they are asked to choose a multitude of 
goals, a large number of which address things other than reading. Consequently, if a student is 
working on just math and is asked about his/her reading their answer would be, "it hasn't". 

I wanted the new form to not only be more generalized in asking about goal achievement, but I 
wanted to force the student to give a specific example both in their tutoring session and outside of 
it. To address this I created the two questions, "Tell me about something you did in your lessons 
which made you proud or happy." "Tell me about something you did at home, work or in a social 
situation which you could not do before your lessons started." 

In my research I uncovered 26 completed Post-Test Evaluation forms from November 1, 
1997 to March 31, 1998. During the course of the project 20 new post-test evaluation forms were 
completed. To evaluate both the number of answers to the questions and the quality of the 
answers, I created a chart which evaluated those questions which addressed goal achievement. In 
one column I listed the accomplished goals from the old Post-Test Evaluation form and in the 
second column I listed those from the new Post-Test Evaluation form. 

My initial evaluation of the forms compared the number of students who mentioned goal 
achievement. The evaluation of the base line forms revealed that four people indicated no goal 
achievements and 22 people had some sort of achievement, a percentage of 84%. On the new form 
only two people listed no goal achievement and 18 had achieved a short term goal, a percentage of 
90%. While not huge, this was a fairly significant increase. 

Many of the students on both forms noted more than one accomplishment. So I compared 
the total number of goals achieved on both forms and discovered that on the old form there were 35 
separate goals mentioned. On the new form there were 33. This translates into a 30% higher rate of 
accomplished goals mentioned on the new Post-Test Evaluation form. 

The questions on both the old Post-Test Evaluation form and the new were open-ended, 
but I believe the new form forced the student to think about at least one specific event which was 
positive. 

For example on the old form several of the answers were, "Improvement. Staff is very 
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supportive of her learning goals." "Reading better - more confidence." "I understand that there are 
dilferent ways to organize information." and, "Came a long way. Starting to understand what she 
is reading." 

Sample answers on the new form were, "Doing [writing] reports at work. Is more detailed 
with those reports." "I read better to my grandchildren." "Using scrap paper and pen at the grocery 
store instead of a calculator." 

The next area I evaluated was a comparison of the change in student goals. On both forms 
we asked the students if they had any change in goals. On the old form the question which 
addressed this was, "Have your goals changed since you entered our program? If so how?" On the 
new form I changed this question to read, "When you began our program you said you wanted to 

. Are you still working towards that goal or has it changed." I then added two other 

questions to determine what, if any, the new long term and short term goals were (Appendix II see 
Question 3a. 3b.). 

This area of the form showed the greatest improvement. On the old Post-Test Evaluation 
form, 21 out of the 26 students indicated no change in their goals. Four of them indicated a 
change, but the change was just more of what they were already working on or was very vague. 
For example, "Yes, wants to do more writing and more reading." "No longer interested in .." 
"Feels better about herself Wants to be able to read and write better." 

On the new Evaluation form there were 8 students who indicated no change in their goals 
and eleven who noted a new goal. This was a fairly significant number and showed that the old 
question was not eliciting enough response from the students to determine if we were still meeting 
their needs. The new goals mentioned were also much more specific. They included, "I want to 
learn more about the internet and open a web page." "Probably get a job." "Check writing." "Learn 
to write better." 

The next section of the form evaluated was the Tutoring Section (Appendix II, #1-6 under 
subheading Tutoring). This section was created to determine if the students were happy with their 
tutor and the time and place of their tutoring. There was no specific question which addressed this 
on the old form, but there was the question, "Are there things you would like to work on that 
aren't being covered in your sessions?" which was designed somewhat to see if the tutors were 
meeting the students needs. The comparison of these two questions was objective. In actuality the 
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question on the old form elicited longer responses from the students and some specific things they 
would like to work on further, "driver's license" "more spelling" "check writing", but did not 
really uncover if the tutor was not meeting the students needs. 

The questions on the new form more specifically addressed the student's happiness with 
their tutor and tutoring sessions, unfortunately these questions did not achieve the desired result. 

Of the six questions all but number 3, "What are you doing in your lessons that is helping you 
achieve your goal?", elicited only yes answers from the students. The questions were obviously 
not open ended enough and this was revealed also by the people who evaluated the form. One 
evaluator indicated "It sounds like you're 'checking up' on the tutor. I'm not sure I'd want to say 
anything negative. I'd probably answer yes, even though my answer might really be no." 

Other than a number of students who answered the questions with an enthusiastic "Oh 
yes!" in response to their happiness with their tutor I feel the evaluator was quite correct in her 
assessment. In all questions other than number 3 the answers were simply "yes" with no more than 
one or two of the 20 students elaborating on that. Yet again, only one of the twenty students had a 
negative comment in answer to all 6 of the questions, and that was in response to a change in the 
hours of the meeting facility, "Millie was, [happy] but is not happy with the Glen Hazel Reading 
Room's new time." 

The final section of this form evaluated was "GPLC Program" (Appendix II questions 1-3 
under GPLC Program). There was nothing similar to this on the old Post-Test Evaluation form. 

My goal here was to determine if the students had some outside issues interfering in their tutoring 
for which we could provide intervention or a social service referral. Upon evaluation, it appeared 
that these questions were also not open-ended enough and I'm not sure the students completely 
understood the nature of the first question, "Is this tutoring what you thought it would be? Please 
explain". 

Seventeen out of twenty of the answers to this questions were, "yes" or "better" [than they 
thought]. The second and third questions more directly addresses student life problems and asked 
if we could intervene. The questions were, "Are there any obstacles in your life that will prevent 
you from achieving your goal? If yes, what are then?" and "How can GPLC help you overcome 
those obstacles." In answer to the first question 16 of the 20 students said "No". Only one of 
twenty students noted a problem that was able to be addressed. This student was having baby- 



sitting problems and the area coordinator called the student's caseworker. The other four seemed to 
feel it was their burden to fix their own problems even though they did not have the knowledge of 
how to do that, "I don’t know." was one response. 

I think the evaluators of these forms said it best when they wrote, "Would you tell a 
stranger about personal problems in your life?" and "Most students answered no — as if to say 
retention I attendance problem’s that they'd possibly had were their own doing or responsibility." 

VI. REFLECTION 

Upon reflection I believe this form was in need of a real change and several of the changes 
made showed a solid improvement over the old form. On the other hand a number of the questions 
I thought were open-ended turned out not to be. These should either be changed or eliminated from 
the form. 

The goals portion of the form showed good results when looking at student goal 
achievement and a change in student goals. 1 would want to keep these questions, although the 
comments by the Reading and Education Specialist that all the questions should, "follow a 
'conversational context'" and "ask it in a conversational manner" may be worth considering. 

I agreed with the evaluators that the questions about the time and place of tutoring and the 
quality of the tutoring were too easy to answer yes and no. I think we should continue to explore 
ways to uncover tutoring problems, but these questions will have to be more open-ended and less 
direct, less like we are "checking up" on the tutor. 

Finally the GPLC program questions regarding life obstacles to tutoring need to be 
changed. We want to be able to provide intervention to students who are in danger of dropping out 
of the program, but these questions are not allowing the students to open up to the evaluator. 
Perhaps a more conversational opening such as "Tell me what's going on in your life." would 
work better here. 

All the evaluators also indicated that although the form was only one page, it was too long. 

I would look at shortening it. 

I do intend to revise this form again based on the results of this project. I would like to get 
more Area Coordinator input on the new form and try it out for several months again. I could do a 
second cycle of research on the newly revised form, but I think we have enough data from this 
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project to complete it. 

In reviewing this form I began to consider all the forms and the large amount of paper 
being used in all Area Coordinator offices. A potential new project could be to look at the paper 
flow in oUr offices and the necessity of many of the forms we use. Do we still need them, and if 
yes, should they be updated and revised. 
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APPENDIX I 
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EVALUATION SHEET 



(The interviewer should complete this form for the student) 



1 . In what ways do you think your reading has changed since you started with your tutor? 



2a. Have you reached any of your own personal reading goals? Please explain and be as specific as you can. 



2b. Have your goals changed since you entered our program? If so, how? 



3. Are your lessons with your tutor benefiting you on a day to day basis either at home or on your job? Please 
explain. 



4. Are there things you would like to work on that aren’t being covered in your sessions? 



3 

ERIC 



APPENDIX II 



Post-Test Evaluation Sheet 

(The interviewer should complete the form for the student) 

Student Name: Date: 

Interviewer: 

Goals 

1 . Tell me about something you did in your lessons which made you proud or happy. 

2. Tell me about something you did at home, work or in a social situation which you could not do before your 
lessons started. 

3. When you began our program you said you wanted to . Axe you still working 

towards that goal or has it changed? 

3 a. If it has changed what do you want to achieve now? 

3b. What new achievements do you want to reach? 

4. Is there anything else you feel you have achieved as a result of your tutoring? 

Tutoring 

1 . Are you happy with the time and place of your tutoring? 

2. Does your tutor show up on time and/or call if she/he has to cancel? Are missed sessions rescheduled? 

3. What are you doing in your lessons that is helping you achieve your goal? 

4. Does your tutor give you clear directions and ask you questions? 

5. Do you feel comfortable asking your tutor questions? 

5 a. Do you understand the answers? 

6. Do you tell your tutor when you don’t understand a lesson? 

GPLC Program 

1. Is this tutoring what you thought it would be? Please explain 

2. Are there any obstacles in your life that will prevent you from achieving your goal? If yes, what are they? 

3. How can GPLC help you overcome those obstacles? 

ERIC 
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APPENDIX III 

Evaluation of the Post-Test Evaluation Sheet 



Please answer the following questions as completely as possible. 

The goal of the New Post-Test Evaluation Sheet was three-fold: 

The first was to uncover student accomplishments and completed short and long term goals. 

The secofid was to uncover any student problems which are occurring in the tutoring. 

The third was to uncover any problems in the students’ personal life which ccndd affect their future tutoring. 

Goals 

1. By using this form were you able to uncover any goals the students had completed which you were 
previously unaware of? Please explain or give an example. 



2. Did you discover that some student’s goals had changed from the pre-test or previous post-test? Were you 
aware of these changes prior to using this form? 



Tutoring: 



1 . By using this form were you able to uncover any problems in the tutoring situation which you were 
previously unaware of? Please explain or give an example. 



GPLC Program 

1 . By using this form, did you discover if students thought the tutoring was not what they expected? If yes, 
were they happy or upset about this? Please explain or give an example. 
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2. By using this form did you uncover any obstacles in the student’s lives which would affect their tutoring? If 
no, was it because the students had no problems or because they were unwilling to reveal problems? If yes, 
were you able to help that student? Please explain. 



The Form 



* I have attached a blank form to this evaluation sheet. Please feel free to write directly on it. 
1 . What did you like best about this form? 



2. What changes would you make? 



3. Do you feel it accomplished any of three stated goals? 



4. If no, what did it not accomplish? If yes, what did it accomplish? 



5. Is there any other goal you would like to see this form accomplish? What is it? 



6. Did you feel the form was too long? Too short? Did it take up too much time? 



7. Any other comments: 
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